Auto-focus system for a digital imaging device and method

ABSTRACT

The present invention relates to an auto-focus system and method based on a sensor, using hierarchical extraction and motion estimation for an image. The auto-focus method may include the steps of: (a) extracting a phase difference image from a sensed image; (b) extracting a robust feature image which satisfies a preset reference with respect to noise and out-of-focus blur, from the phase difference image; and (C) estimating a motion vector for lens shift from the feature image.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority of Korean Patent Application No.10-2015-0180365, filed on Dec. 16, 2015, which is incorporated herein byreference in its entirety.

BACKGROUND

1. Field

Exemplary embodiments of the present invention relate generally to anauto-focus technology for a digital imaging device and moreparticularly, an auto-focus system and method based on a sensor usinghierarchical feature extraction and motion estimation for an image.

2. Description of the Related Art

The auto-focus technology is one of the most important technologies foracquiring a clear image in an image acquisition process of a digitalimaging device, such as a camera.

The auto-focus technology may be generally classified in active methodswhich directly measure a distance using an additional signal, andpassive methods which measure a distance by analyzing light transmittedthrough a lens.

Active methods generally employ a module for auto-focus, and calculate afocal distance based on the time required for a specific signal which isemitted by the camera to return to the camera after it is reflected onan object. The passive method calculates a distance by analyzing a lightsource of a visible area, which is reflected from an object andtransmitted to the camera.

The passive method may be generally divided into a Contrast DetectionAuto-Focus (CDAF) method and a Phase Detection Auto-Focus (PDAF) method.All of the auto-focus methods are currently applied to digital cameras.

Recently, much attention has been paid to a hybrid auto-focus methodwhich combines the advantages of the active method and the passivemethod.

The hybrid auto-focus method primarily determines an in-focus stateusing the active method or the PDAF method, and secondarily provides aclear in-focus image using the CDAF method which generally may providebetter precision.

In the hybrid auto-focus method, the primary auto-focus method is veryimportant in determining the computing speed of the auto-focus system.That is because, in the primary auto-focus method, the lens needs to beas close as possible to the in-focus state, in order to minimizerepetitive motions of the lens in the CDAF method which is used as thesecondary auto-focus method.

For the hybrid auto-focus method, research has been recently conductedon a focus method in which pixels for calculating a phase difference aremounted in an imaging sensor.

In this method, two measuring point pixels used for auto-focus arecovered with a black mask so as to receive different phases.

However, due to the black mask which is separately mounted on the pixelsin order to define different phases, the amount of light received by themeasuring point pixels is smaller than the amount of light received byother pixels, and out-of-focus blur exists in an image inputted to thecamera.

Under such a condition, it is very difficult to detect a feature and toacquire a reliable phase difference. Thus, there is a demand for amethod which is capable of extracting a reliable phase difference image,calculating an accurate phase difference through the phase differenceimage, and estimating a motion.

Japanese Patent Laid-Open Publication NO. 2014-182237, Japanese PatentLaid-Open Publication NO. 2013-152388, US Publication NO. 2013-0271646,US Publication No. 2013-0235253 and US Publication No. 2012-0133821describe generally autofocus systems for digital cameras.

SUMMARY

Various embodiments are directed to an auto-focus system and methodbased on a sensor, which is capable of extracting a reliable phasedifference image.

Also, various embodiments are directed to an auto-focus system andmethod based on a sensor, which is capable of estimating a motion byextracting a strong feature from a phase difference image.

Also, various embodiments are directed to an auto-focus system andmethod based on a sensor, which is capable of minimizing repetitivemotions of a lens for in-focused image through motion estimation.

In an embodiment, an auto-focus system may include: a sensing unitsuitable for sensing an image; and an auto-focus control unit suitablefor extracting a phase difference image from the image, extracting astrong feature image from the phase difference image, and estimating amotion vector for lens shift from the feature image.

The auto-focus control unit may be suitable for extracting the featureimage satisfying a preset reference with respect to noise andout-of-focus blur.

The auto-focus control unit may include: a phase difference imageextraction unit suitable for selecting a predetermined interest area inthe image, and extracting first and second phase difference images inresponse to an image of the interest area and a predefined measuringpoint pattern image; a feature extraction unit suitable for acquiring afirst difference image and a second difference image in response to anyone of the first and second phase difference images, acquiring thecoordinate of a predefined area using the first and second differenceimages, and extracting the feature image corresponding to the coordinatefrom the first and second phase difference images; and a motionestimation unit suitable for estimating the motion vector in the featureimage.

The phase difference image extraction unit may include: an interest areasetting unit suitable for setting the interest area having a predefinedposition and size in the image; an operation unit suitable for taking animage of an measuring point position based on the image of the interestarea and the measuring point pattern image; and an image sampling unitsuitable for acquiring the first and second phase difference images bysampling the image of the measuring point position.

The feature extraction unit may be suitable for acquiring the firstdifference image being resistant to noise and the second differenceimage being resistant to out-of-focus blur.

The feature extraction unit is suitable for acquiring the coordinate ofthe predefined area having a strong feature using the first and seconddifference images.

The feature extraction unit may include: a Difference of Gaussian (DoG)unit suitable for acquiring the first difference image by applying aDifference of Gaussian (DoG) scheme to the first phase difference image;a multi-scale image difference unit suitable for acquiring the seconddifference image by applying a difference scheme between imagesreconstructed by a pyramid method to the first phase difference image;an arithmetic unit suitable for summing the first difference image andthe second difference image and acquiring the coordinate of thepredefined area; and a feature image generation unit suitable forextracting the feature image including first and second feature imagescorresponding to the coordinate from the first and second phasedifference images.

The motion estimation unit may estimate the motion vector using at leastone of phase correlation, block matching, and hierarchicalinterpolation, in response to the feature image.

In an embodiment, an auto-focus method may include: extracting a phasedifference image from an image; extracting a strong feature image fromthe phase difference image; and estimating a motion vector for lensshift from the feature image.

The step extracting of the phase difference image may include: selectinga predefined interest area in the image; and extracting first and secondphase difference images in response to an image of the interest area anda predefined measuring point pattern image.

The step extracting of the phase difference image may include:reconstructing the image into an image of an interest area having apredefined position and size; acquiring an image of an measuring pointposition based on the image of the interest area and the measuring pointpattern image; and acquiring the first and second phase differenceimages by sampling the image of the measuring point position.

The extracting of the feature image may include extracting the featureimage which satisfies a preset reference with respect to noise andout-of-focus blur, from the phase difference image.

The extracting of the feature image may include: acquiring a firstdifference image by applying a Difference of Gaussian (DoG) scheme tothe first phase difference image; acquiring a second difference image byapplying a difference scheme between images reconstructed by a pyramidmethod to the first phase difference image; acquiring the coordinate ofthe predefined area strong based on the first and second differenceimages; and extracting the feature image including first and secondfeature images corresponding to the coordinate from the first and secondphase difference images.

The estimating of the motion vector may include estimating the motionvector in the feature image, using at least one of phase correlation,block matching, and hierarchical interpolation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B are diagrams illustrate a conventional phase differenceauto-focus method.

FIGS. 2A, 2B and 3 are diagrams illustrating the structure of a sensorwhich is applied to an auto-focus system, according to an embodiment ofthe present invention.

FIG. 4 is a block diagram illustrating an auto-focus system, accordingto an embodiment of the present invention.

FIG. 5 is a block diagram illustrating a phase difference imageextraction unit.

FIG. 6 is a diagram illustrating an extraction process of a featureextraction unit, according to an embodiment of the present invention.

FIG. 7 is a diagram illustrating a process of generating a phasedifference image, according to an embodiment of the present invention.

FIG. 8 is a diagram illustrating an hierarchical interpolation formotion estimation, according to an embodiment of the present invention.

DETAILED DESCRIPTION

Hereinafter, various embodiments of the invention including anauto-focus system and method for a digital camera will be described indetail with reference to the accompanying drawings.

While describing the present invention, detailed descriptions related topublicly well-known functions or configurations will not be repeatedherein in order not to unnecessarily obscure the subject matter of thepresent invention.

The terms such as first and second can be used to describe variouselements. However, the components are not limited by these terms, andthese terms are used only to distinguish one element from anotherelement.

FIGS. 1A and 1B are diagrams illustrating a conventional phasedifference auto-focus method for a digital camera.

FIG. 1A illustrates a process in which a light source transmittedthrough a lens is split into two lights formed in an imaging sensor anda line sensor, respectively, through a half mirror and one of asub-mirror or a mirror.

FIG. 1B illustrates a process in which light is separated through aseparate lens and formed in a sensor. When out-of-focus blur exists, asymmetrical image is formed with respect to the center of each linesensor.

The Phase Detection Auto-Focus (PDAF) method calculates a phasedifference through a line sensor and adjusts a focus according to thephase difference.

Most Digital Single Lens Reflex (DSLR) cameras employ a separate linesensor for calculating a phase difference through a signal of the linesensor, and estimate the direction and extent of focusing in response tothe phase difference.

Since the PDAF method calculates a phase difference through a lightsource formed on a separate line sensor, the PDAF method may rapidlydetermine the in-focus position of the lens with respect to a targetobject.

However, the PDAF method requires many separate devices for transmittinglight to the line sensor. When the hardware structure, such as the halfmirror or the separate lens, is varied from the initial structure due toan external impact, the PDAF method cannot acquire an accurate focus.

FIGS. 2A, 2B and 3 are diagrams illustrating the structure of a sensor 1which is applied to an auto-focus system, according to an embodiment ofthe present invention.

Referring to FIGS. 2A, 2B and 3, the sensor 1 according to theembodiment of the present invention, may use all pixels as measuringpoints for detecting a phase difference.

As illustrated in FIGS. 2A and 2B, the sensor 1 may include a photodiode2, a black mask 4, and a microlens 6 in each pixel. The photodiode 2 isutilized as a measuring point for receiving a light source, the blackmask 4 is installed in front of the photodiode 2, and the microlens 6 isinstalled in front of the photodiode 2 and the black mask 4.

Referring to FIG. 3, the sensor 1 according to the embodiment of FIG. 2may utilize all pixels as measuring points for acquiring a phase image.For example, the pixels may be arranged in a Bayer Color Filter Array(CFA) Blue, Green, Green, Red (BGGR) structure. Among the pixels, anodd-numbered pixel 3A in the horizontal direction may include aphotodiode 2 arranged in the left side thereof and a black mask 4arranged in the right side thereof, and an even-numbered pixel 3B in thehorizontal direction may include a photodiode 2 arranged in the rightside thereof and a black mask 4 arranged in the left side thereof.

An image obtained through a camera including the sensor 1 with theabove-described structure may be defined as the following out-of-focusblur model.

$\begin{matrix}{{{g\left( {x,y} \right)} = {{g_{L}\left( {x,y} \right)} + {g_{R}\left( {x,y} \right)}}},{{where}\mspace{14mu}\left\{ {\begin{matrix}{{g_{L}\left( {x,y} \right)} = {{{f\left( {x,y} \right)}*{h_{L}\left( {x,y} \right)}} + {\eta\left( {x,y} \right)}}} \\{{g_{R}\left( {x,y} \right)} = {{{f\left( {x,y} \right)}*{h_{R}\left( {x,y} \right)}} + {\eta\left( {x,y} \right)}}}\end{matrix},} \right.}} & (1)\end{matrix}$

In Equation 1, g(x, y) represents an acquired image, f(x, y) representsan ideal image with no out-of-focus blur, acquired from a Bayer pattern,g_(L)(x, y) and g_(R)(x, y) represent left and right phase images without-of-focus blur, respectively, h_(L)(x, y) and h_(R)(x, y) representthe focus degradation functions of g_(L)(x, y) and g_(R)(x, y),respectively, η(x, y) represents a noise image, and (x, y) representsthe coordinate of the image.

FIG. 4 is a block diagram illustrating an auto-focus system, accordingto an embodiment of the present invention.

Referring to FIG. 4, the auto-focus system, according to an embodimentof the present invention, includes a sensing unit 100 and an auto-focuscontrol unit 200.

The sensing unit 100 senses an image and provides the sensed image tothe auto-focus control unit 200. The auto-focus control unit 200extracts a phase difference image from the image provided from thesensing unit 100, extracts a feature image resistant to noise andout-of-focus blur from the phase difference image, and estimates amotion vector for lens shift from the feature image.

The auto-focus control unit 200 may include a phase difference imageextraction unit 10, a feature extraction unit 20, and a motionestimation unit 30.

The phase difference image extraction unit 10 generates left and rightphase images g_(L)(x, y) and g_(R)(x, y) in response to a defocusedimage g(x, y) and an image b(x, y) defining measuring points. At thistime, b(x, y) represents a predefined measuring point pattern image, andg_(L)(x, y) and g_(R)(x, y) represent extracted left and right phaseimages, respectively.

The feature extraction unit 20 redefines interest areas of two phaseimages based on a strong feature point extracted from one of the phaseimages (e.g., the left phase image), and extracts strong feature pointsfrom the noise of an image and a defocused image, using the Differenceof Gaussian (DoG) algorithm and a difference scheme between imagesreconstructed through a pyramid method in image processing.

The feature extraction unit 20 may include a Difference of Gaussian(DoG) unit 22, a multi-scale image difference unit 24, an arithmeticunit 28, and a feature image generation unit 26.

The DoG unit 22 acquires a difference image resistant to noise byapplying the DoG algorithm to the left phase difference image. Themulti-scale image difference unit 24 acquires a difference imageresistant to out-of-focus blur by applying the difference scheme toimages reconstructed through a pyramid method. The arithmetic unit 28sums the difference images provided from the DoG unit 22 and themulti-scale image difference unit 24, and acquires the coordinate of anarea with a strong feature. The feature image generation unit 26extracts feature images from the phase difference images, based on thecoordinate of the area with a strong feature.

The motion estimation unit 30 calculates a motion vector (Δx, Δy) inresponse to phase correlation between two images g_(Lf)(x, y) andg_(Rf)(x, y) acquired as the feature extraction result by the featureextraction unit 20. The motion vector (Δx, Δy) is used to shift a lensfor acquiring an in-focused image.

The auto-focus system and method according to an embodiment of thepresent invention will be described in more detail as follows. First,the process of acquiring a phase difference image from an image acquiredthrough the sensor will be described.

FIG. 5 is a block diagram illustrating an example of the phasedifference image extraction unit 10 of FIG. 4 according to an embodimentof the invention.

Referring to FIG. 5, the phase difference image extraction unit 10 mayinclude an interest area setting unit 12, an operation unit 14, and animage sampling unit 16.

The interest area setting unit 12 sets the image g(x, y) sensed throughthe sensing unit 100 to the position and size of a predefined interestarea. The operation unit 14 performs an AND operation on the image ofthe interest area and the measuring point pattern image b(x, y). Theimage sampling unit 16 acquires phase difference images g_(L)(x, y) andg_(R)(x, y) by sampling an output image of the operation unit 14.

The phase difference image extraction unit 10 generates the left phaseimage g_(L)(x, y) from the pixels in the odd-numbered columns, based onthe Bayer CFA BGGR structure of the sensor of the sensing unit 100. Thephase difference image extraction unit 10 also generates the right phaseimage g_(R)(x, y) from the pixels in the even-numbered columns, based onthe Bayer CFA BGGR structure of the sensor of the sensing unit 100. Atthis time, b(x, y) represents the predefined measuring point patternimage, and g_(L)(x, y) and g_(R)(x, y) represent the extracted left andright phase images, respectively.

As described above, the phase difference image extraction unit 10reconstructs the image g(x, y) acquired through the sensor based on theposition and size of the predefined interest area, and acquires onlyinformation of the measuring point positions by performing an ANDoperation on g(x, y) and b(x, y). Furthermore, the phase differenceimage extraction unit 10 defines the x and y axes of the respectivemeasuring points, and extracts the left and the right phase imagesg_(L)(x,y) and g_(R)(x, y) by sampling the image of the operation unit14.

Next, the process of extracting a strong feature from the phasedifference images will be described.

When an image is taken through the sensor having the structureillustrated in FIGS. 2 and 3, the amount of light transmitted to thesensor decreases. Thus, the image g(x, y) acquired through the sensor isa low-luminance image which has lower brightness than the ideal imagef(x, y) with no out-of-focus blur in the Bayer pattern.

Therefore, the left phase image g_(L)(x, y) and the right phase imageg_(R)(x, y) are also low-luminance images, and the first derivativevalues of the two images are smaller than the first derivative value ofideal image f(x, y). Furthermore, when the ISO sensitivity is raised inthe low-luminance environment, the proportion of noise increases in g(x,y) due to the influence of η(x, y). When an image reflected from anobject is not an in-focused image, the image g(x, y) has out-of-focusblur due to convolution with h(x, y).

For solving the above-described problem, the feature extraction unit 20performs DoG on g_(L)(x, y) to obtain a high-frequency componentresistant to noise. The result obtained by performing DoG may beexpressed as Equation 2 below.

$\begin{matrix}{{D_{\sigma_{i},\sigma_{i + 1}}\left( {x,y} \right)} = {{\sum\limits_{u = {- k}}^{k}\;{\sum\limits_{v = {- s}}^{s}\;{{h_{\sigma_{i}}\left( {u,v} \right)}{g_{L}\left( {{x + u},{y + v}} \right)}}}} - {\sum\limits_{u = {- k}}^{k}\;{\sum\limits_{v = {- s}}^{s}\;{{h_{\sigma_{j}}\left( {u,v} \right)}{g_{L}\left( {{x + u},{y + v}} \right)}}}}}} & (2)\end{matrix}$

In Equation 2, σ_(i) represents a size variable of the Gaussianfunction, represents a coefficient of a Gaussian kernel, and k and srepresents neighboring pixels of x and y. Furthermore, D_(σ) _(i) _(,σ)_(i+1) (x,y) is normalized to a value between 0 and 0.5, in order tosatisfy Equation 3 below.

$\begin{matrix}{{{\overset{\sim}{D}}_{\sigma_{i},\sigma_{i + 1}}\left( {x,y} \right)} = {\frac{{D_{\sigma_{i},\sigma_{j}}\left( {x,y} \right)} - {\min\left\{ {D_{\sigma_{i},\sigma_{j}}\left( {x,y} \right)} \right\}}}{{\max\left\{ {D_{\sigma_{i},\sigma_{j}}\left( {x,y} \right)} \right\}} - {\min\left\{ {D_{\sigma_{i},\sigma_{j}}\left( {x,y} \right)} \right\}}} \times 0.5}} & (3)\end{matrix}$

In order to extract a feature resistant to out-of-focus blur, thefeature extraction unit 20 uses a difference image between two imageswith different size variables, the difference image being acquired bydown-sampling the two images and then up-sampling the down-sampledimages. The difference image between the two images with different sizevariables is defined as follows.

$\begin{matrix}{{S_{\kappa_{i}\kappa_{j}}\left( {x,y} \right)} = {{\frac{1}{\kappa_{i}^{2}}{\sum\limits_{u = 0}^{\kappa_{i} - 1}\;{\sum\limits_{v = 0}^{\kappa_{i} - 1}\;{g_{L}\left( {{{x/\kappa_{i}} + u},{{y/\kappa_{i}} + v}} \right)}}}} - {\frac{1}{\kappa_{j}^{2}}{\sum\limits_{u = 0}^{\kappa_{j} - 1}\;{\sum\limits_{v = 0}^{\kappa_{j} - 1}\;{g_{L}\left( {{{x/\kappa_{j}} + u},{{y/\kappa_{j}} + v}} \right)}}}}}} & (4)\end{matrix}$

In Equation 4, k_(j) represents a size variable, and S_(K) _(i) _(,K)_(j) (x, y) represents the result image. Since an arithmetic mean filterwas applied according to the sampling size variable, noise of S_(K) _(i)_(,K) _(j) (x,y) is reduced, and S_(K) _(i) _(,K) _(j) (x,y) has strongedge information through repetitive sampling. Furthermore, S_(K) _(i)_(,K) _(j) (x,y) is also normalized as follows.

$\begin{matrix}{{{\overset{\sim}{S}}_{\kappa_{i},\kappa_{j}}\left( {x,y} \right)} = {\frac{{S_{\kappa_{i},\kappa_{j}}\left( {x,y} \right)} - {\min\left\{ {S_{\kappa_{i},\kappa_{j}}\left( {x,y} \right)} \right\}}}{{\max\left\{ {S_{\kappa_{i},\kappa_{j}}\left( {x,y} \right)} \right\}} - {\min\left\{ {S_{\kappa_{i},\kappa_{j}}\left( {x,y} \right)} \right\}}} \times 0.5}} & (5)\end{matrix}$

The feature extraction unit 20 detects the most strong edge component bysumming the above-described two images. The image obtained by summingthe two images may be defined as follow.p _(L)(x,y)={tilde over (D)} _(σ) _(i) _(,σ) _(j) (x,y)+{tilde over (S)}_(K) _(i) _(,K) _(j) (x,y)  (6)

In Equation 6, p_(L)(x, y) represents a result obtained by addingbrightness values. Since {tilde over (D)}_(σ) _(i) _(,σ) _(j) (x,y) and{tilde over (S)}_(K) _(i) _(,K) _(j) (x,y) were normalized to a valuebetween 0 and 0.5 in Equations 3 and 5, p_(L)(x, y) has a value between0 and 1.

At this time, {tilde over (D)}_(σ) _(i) _(,σ) _(j) (x,y) has an effectof suppressing noise in a low-luminance image, but has difficulties inextracting a feature point from a defocused image. On the other hand,{tilde over (S)}_(K) _(i) _(,K) _(j) (x,y) is suitable for extracting afeature point from a defocused image, but is also relatively vulnerableto noise.

Thus, a pixel having a brightness value close to 1 in p_(L)(x, y) may beset to be a feature point. Finally, as the coordinate of an area havinga strong feature is acquired from p_(L)(x, y), an image g_(Lf)(x, y) iscut and acquired from g_(L)(x, y) based on the acquired coordinate. Theimage g_(Rf)(x, y) is also acquired from the image g_(R)(x, y), based onthe acquired coordinate.

FIG. 6 is a diagram illustrating the feature extraction process of thefeature extraction unit 20 of FIG. 4, according to an embodiment of theinvention The auto-focus system according to an embodiment of thepresent invention extracts a feature image by adjusting size variablesσ_(i) and k_(j) of the Gaussian function.

Referring to FIG. 6, the feature extraction unit 20 adjusts the sizevariable k_(j), and extracts hierarchical down-sampled images fromg_(L)(x, y). Since S_(K) _(i) _(,K) _(j) (x,y) is a difference resultbetween two images having different size variables K_(j), K_(j) andK_(j+1) may have a relation of an integer multiple. In the presentembodiment, a relation of 2K_(j)=K_(j+1) is defined.

The feature extraction unit 20 extracts hierarchical blurred images fromg_(L)(x, y) according to the size variable σ_(i) of the Gaussianfunction. Furthermore, D_(σ) _(i) _(,σ) _(j) (x,y) determines thestrength of the edge according to the difference between σ_(i) andσ_(i+1). In the present embodiment, a relation of 5σ_(i)=σ_(i+1) may beestablished in order to detect a strong edge.

FIG. 7 is a diagram illustrating a process of generating a phasedifference image, according to an embodiment of the invention.

Referring to FIGS. 3 and 7, when the black mask 4 is installed in theright side of the photodiode 2, the light source is gathered in the leftside of the photodiode 2. When the black mask 4 is installed in the leftside of the photodiode 2, the light source is gathered in the right sideof the photodiode 2.

When the initial input image g(x, y) is sampled according to themeasuring points, two phase images are obtained, the two phase imageshaving horizontal axis information which is symmetrically spread due tothe black masks, and high-frequency components of the two phase imageshave a difference with respect to the horizontal axis. In the case ofthe vertical axis, however, light is formed in the photodiodes acrossthe whole vertical components of the pixels. Therefore, a phasedifference caused by the black mask 4 does not exist between the twophase images.

As illustrated in 710 of FIG. 7, light reflecting from an object isinputted to the camera. As illustrated in 720 of FIG. 7, two imagesg_(L)(x, y) and g_(R)(x, y) degraded by the focus degradation functionsh_(L)(x, y) and h_(R)(x, y) are sampled in the odd-numbered columns andthe even-numbered columns of the sensor 1, respectively.

As illustrated in 730 of FIG. 7, the angle at which the object isobserved by the pixels of the odd columns is different from the angle atwhich the object is observed by the pixels of the even columns. Thus, aphase difference occurs according to the respective focus states.Furthermore, since the measuring point pairs of the sensor 1 arepositioned at an interval of one pixel, a decimal phase differenceoccurs.

The auto-focus system according to an embodiment of the presentinvention may use one of phase correlation and block matching, in orderto estimate a motion vector, and uses hierarchical interpolation inorder to perform decimal calculation.

First, the use of phase correlation for motion estimation will bedescribed.

In order to calculate the power spectrum of a feature image, two featureimages are Fourier-transformed, and the cross correlation therebetweenis calculated. The cross correlation between the two feature imageswhich are Fourier-transformed is defined as follows.

$\begin{matrix}{{E_{PC}\left( {\mu,v} \right)} = \frac{{??}\left\{ {g_{Rf}\left( {x,y} \right)} \right\} \times {??}\left\{ {g_{Lf}^{*}\left( {x,y} \right)} \right\}}{{{??}\left\{ {g_{Rf}\left( {x,y} \right)} \right\} \times {??}\left\{ {g_{Lf}^{*}\left( {x,y} \right)} \right\}}}} & (7)\end{matrix}$

In Equation 7, (μ, ν) represents a frequency spatial coordinatecorresponding to (x, y), g*_(Lf)(x,y) represents a conjugate result ofg_(Lf)(x,y), ℑ{g_(Rf)(x,y)} and ℑ{g*_(Lf)(x,y)} represent Fouriertransform results of g_(Rf)(x,y) and g*_(Lf)(x,y), respectively, andE_(PC)(μ,ν) represents a result obtained by calculatingcross-correction. Then, E_(PC)(μ,ν) is reverse Fourier transformed, andthe maximum point is acquired as a motion vector.

The motion vector is defined as follows.(Δx,Δy)=max[ℑ{E _(PC)(μ,ν)}]  (8)

In Equation 8, ℑ⁻¹{E_(PC)(μ,ν)} represents a result obtained byreverse-Fourier-transforming E_(PC)(μ,ν), and (Δx, Δy) represents aphase difference between two phase images. The result ofℑ⁻¹{E_(PC)(μ,ν)} indicates the extent of motion at the spatialcoordinate (x, y). Thus, the final motion coordinate of the two imagesbecomes the position at which ℑ⁻¹{E_(PC)(μ,ν)} has the largest value.That is, when high-frequency components of the two images exist at thesame pixel position, a relation of (Δx, Δy)=(0, 0) is established.

Next, the use of block matching for motion estimation will be describedas follows.

The block matching method measures a displacement by evaluating spatialcoordinates on a block basis, unlike the phase correlation method. Theblock matching method compares the brightness values of pixels includedin a block of one image to another image, selects the block of thecompared image, in which the compared pixels have the smallest error,and thus estimates the motions of two blocks.

The block matching method may be divided into three steps. First, anevaluation method for matching two images is defined. Second, a searchstrategy for a comparison area for moving a block is established.Finally, the size of the block is determined to establish a hierarchicalor adaptive strategy.

The block evaluation method used at the first step includes Minimum meanSquare Error (MSE), Mean Absolute Difference (MAD), and Sum of AbsoluteDifference (SAD). In the present embodiment, the SAD is used in order toguarantee the reliability of the absolute difference of each pixeldifference.

The SAD may be defined as follows.

$\begin{matrix}{{{SAD}\left( {d_{1},d_{2}} \right)} = {\sum\limits_{{({n_{1},n_{2}})} \in B}\;{{{s_{1}\left( {n_{1},n_{2}} \right)} - {s_{2}\left( {{n_{1} + d_{1}},{n_{2} + d_{2}}} \right)}}}}} & (9)\end{matrix}$

In Equation 9, (d₁,d₂) represents a displacement vector of twocomparison images, (n₁,n₂) represents a reference pixel to be compared,s₁(n₁,n₂) represents a reference image, and s₂(n₁,n₂) represents animage to be compared.

When the search strategy is used at the second step, the image of whichthe feature is extracted is defined as blocks while the width and heightof the image are reduced, for example two times, in order to increasethe accuracy of the experiment. Then, the image of which the feature isextracted is set to the search range, in order to search the wholesections. Finally, while the left phase image of the image of which thefeature is extracted is shifted on a basis of 0.001 in order to estimatea decimal displacement vector, hierarchical block matching is performed.

Finally, a hierarchical interpolation for motion estimation will bedescribed.

FIG. 8 is a diagram illustrating the use of hierarchical interpolationfor motion estimation, according to an embodiment of the invention.

Referring to FIG. 8, interpolation is performed to shift the leftfeature image g_(Lf)(x, y) by one pixel in the range of [−5, 5]. Then,the displacement of the shifted image is calculated, and (Δx, Δy) whichis the closest to (0, 0) is searched to estimate the motion of the imageon a basis of one pixel.

When the minimum value of (Δx, Δy) is not (0, 0), the mean of twointerpolated values which are the closest to (0, 0) is acquired as thefinal motion value. Then, the left feature image g_(Lf)(x, y) shifted byan integer number of pixels is interpolated in the range of [−1, 1] on abasis of 0.1, and the minimum value of (Δx, Δy) is calculated.

That is, the decimal motion vector is estimated while the interpolationunit of the image is gradually reduced.

In the sensor, a light source is transmitted to the left areas of thepixels in all odd-numbered columns, and light is formed in the rightareas of the pixels in all even-numbered columns. Thus, in the presentembodiment, the image is shifted only along the x-axis, the motionestimation result also reflects only Δx, and Δy is already defined inthe camera module.

As described above, the auto-focus system according to an embodiment ofthe present invention may detect a feature resistant to low luminanceand out-of-focus blur, use the photodiodes as a sensor for acquiring aphase difference, and acquire the motion results of two phase images inorder to move a length. Thus, the image may become resistant to anexternal impact and an optical axis of light which is reflected from theobject and transmitted to the sensor.

The auto-focus system according to an embodiment of the presentinvention, can more accurately measure the displacement by which thelens is to be moved. Thus, the auto-focus system can acquire anin-focused image resistant to an optical axis and external impact in thelow-luminance environment. In this aspect, the auto-focus system canacquire a phase difference using only pixels covered with a black maskwithout an additional sensor. Thus, the auto-focus system can be mountedin a hybrid auto-focus system and provide convenience during an in-focusimage acquisition process.

According to an embodiment of the present invention, the auto-focussystem and method can extract a reliable phase difference image from alow-luminance defocused image.

Furthermore, since the auto-focus system and method estimates a motionby extracting a strong feature from a phase difference image, theauto-focus system and method can rapidly calculate a phase difference ina low-luminance environment.

Furthermore, since the auto-focus system and method accurately andrapidly calculate a phase difference, the auto-focus system and methodcan minimize repetitive motions of the lens for an in-focus state.

Furthermore, since the auto-focus system and method minimizes repetitivemotions of the lens, the auto-focus system and method can rapidly set anin-focus state.

Although various embodiments have been described for illustrativepurposes, it will be apparent to those skilled in the art that variouschanges and modifications may be made without departing from the spiritand/or scope of the invention as defined in the following claims.

What is claimed is:
 1. An auto-focus system for a digital electronicdevice that adjust focus by lens shift comprising: a sensing unitsuitable for sensing an image; and an auto-focus control unit suitablefor extracting first and second phase images having a phase differencefrom the image sensed by the sensing unit, extracting a strong featureimage from the first and second phase images, and estimating a motionvector used for the lens shift to adjust focus from the feature image,wherein the auto-focus control unit comprises: a feature extraction unitsuitable for acquiring a first difference image and a second differenceimage by applying a Difference of Gaussian scheme and a differencescheme between images reconstructed by a pyramid method to any one ofthe first and second phase images, acquiring coordinate of a predefinedarea by summing the first and second difference images, and extractingthe feature image corresponding to the coordinate from the first andsecond phase images.
 2. The auto-focus system of claim 1, wherein theauto-focus control unit is suitable for extracting the feature imagesatisfying a preset reference with respect to noise and out-of-focusblur.
 3. The auto-focus system of claim 1, wherein the auto-focuscontrol unit further comprises: a phase difference image extraction unitsuitable for selecting a predetermined interest area in the image sensedby the sensing unit, and extracting the first and second phase images bytaking an image of an measuring point position based on an image of theinterest area and a predefined measuring point pattern image andsampling the image of the measuring point position; and a motionestimation unit suitable for estimating the motion vector used for thelens shift to adjust focus from the feature image.
 4. The auto-focussystem of claim 3, wherein the phase difference image extraction unitcomprises: an interest area setting unit suitable for setting theinterest area having a predefined position and size in the image; anoperation unit suitable for taking the image of the measuring pointposition based on the image of the interest area and the measuring pointpattern image; and an image sampling unit suitable for acquiring thefirst and second phase images by sampling the image of the measuringpoint position.
 5. The auto-focus system of claim 1, wherein the featureextraction unit is suitable for acquiring the first difference imagebeing resistant to noise and the second difference image being resistantto out-of-focus blur.
 6. The auto-focus system of claim 1, wherein thefeature extraction unit is suitable for acquiring the coordinate of thepredefined area having a strong feature using the first and seconddifference images.
 7. The auto-focus system of claim 1, wherein thefeature extraction unit comprises: a Difference of Gaussian (DoG) unitsuitable for acquiring the first difference image by applying theDifference of Gaussian (DoG) scheme to the first phase image; amulti-scale image difference unit suitable for acquiring the seconddifference image by applying the difference scheme between imagesreconstructed by a pyramid method to the first phase image; anarithmetic unit suitable for summing the first difference image and thesecond difference image and acquiring the coordinate of the predefinedarea; and a feature image generation unit suitable for extracting thefeature image including first and second feature images corresponding tothe coordinate from the first and second phase images.
 8. The auto-focussystem of claim 1, wherein the motion estimation unit is suitable forestimating the motion vector using at least one of phase correlation,block matching, and hierarchical interpolation, in response to thefeature image.
 9. The auto-focus system of claim 1, wherein the sensingunit comprises a sensor including pixels which have a Bayer Color FilterArray (CFA) pattern, and the sensor is set to use all pixels asmeasuring points for acquiring a phase image.
 10. The auto-focus systemof claim 9, wherein the pixel comprises: a microlens suitable fortransmitting light; a black mask suitable for blocking a part of thelight; and a photodiode suitable for receiving the light of which thepart is blocked by the black mask.
 11. The auto-focus system of claim10, wherein the black mask is arranged to be symmetrical between anodd-numbered pixel and an even-numbered pixel in one direction.
 12. Anauto-focus method for a digital electronic device that adjust focus bylens shift comprising: extracting first and second phase images having aphase difference from an image sensed by a sensing unit; extracting astrong feature image from the first and second phase images; estimatinga motion vector used for the lens shift to adjust focus from the featureimage; and performing the lens shift to adjust focus using the motionvector, wherein the extracting of the feature image comprises: acquiringa first and difference images by applying a Difference of Gaussianscheme and a difference scheme between images reconstructed by a pyramidmethod to any one of the first and second phase images; acquiringcoordinate of a predefined area by summing the first and seconddifference images; and extracting the feature image corresponding to thecoordinate from the first and second phase images.
 13. The auto-focusmethod of claim 12, wherein the extracting of the phase image comprises:selecting a predefined interest area in the image; and extracting firstand second phase images in response to an image of the interest area anda predefined measuring point pattern image.
 14. The auto-focus method ofclaim 12, wherein the extracting of the phase images comprises:reconstructing the image into an image of an interest area having apredefined position and size; acquiring an image of an measuring pointposition based on the image of the interest area and the measuring pointpattern image; and acquiring the first and second phase images bysampling the image of the measuring point position.
 15. The auto-focusmethod of claim 12, wherein the extracting of the feature imagecomprises extracting the feature image which satisfies a presetreference with respect to noise and out-of-focus blur, from the phaseimages.
 16. The auto-focus method of claim 12, wherein the firstdifference image is suitable for being resistant to noise and the seconddifference image is suitable for being resistant to out-of-focus blur.17. The auto-focus method of claim 12, wherein the predefined area is anarea having a strong feature.
 18. The auto-focus method of claim 12,wherein the extracting of the feature image comprises: acquiring a firstdifference image by applying the Difference of Gaussian (DoG) scheme tothe first phase image; acquiring a second difference image by applyingthe difference scheme between images reconstructed by a pyramid methodto the first phase image; acquiring the coordinate of the predefinedarea strong based on the first and second difference images; andextracting the feature image including first and second feature imagescorresponding to the coordinate from the first and second phase images.19. The auto-focus method of claim 12, wherein the estimating of themotion vector comprises estimating the motion vector in the featureimage, using at least one of phase correlation, block matching, andhierarchical interpolation.